Handling unavailable memories in distributed storage network

ABSTRACT

A distributed storage network includes a distributed storage processing unit, which generates multiple data slices from a single data segment, and multiple different storage units to store the data slices. Each data slice is assigned to a particular storage unit for storage. The distributed storage processing unit determines which of the storage units are in a ready state, and sends data slices to the ready storage units. Data slices assigned to storage units that are not in a ready state are temporarily stored in a temporary memory, e.g. cache memory, until the storage unit becomes ready. If the intended storage unit remains in a non-ready status for too long, the slices stored in cache memory can be sent to a substitute storage unit, or discarded and reconstructed at a later time.

CROSS REFERENCE To RELATED PATENTS

This application claims the benefit of U.S. Provisional Application No.61/246,876, filed Sep. 29, 2009, and entitled “DISTRIBUTED STORAGENETWORK MEMORY UTILIZATION OPTIMIZATION,” which is incorporated hereinin its entirety by reference for all purposes.

The present application is related to the following co-pendingapplications:

-   -   1. Utility application Ser. No. 12/777,850 filed on even date        herewith, and entitled “DISTRIBUTED STORAGE NETWORK INCLUDING        MEMORY DIVERSITY”;    -   2. Utility application Ser. No. 12/______ filed on even date        herewith, and entitled “DISTRIBUTED STORAGE NETWORK UTILIZING        MEMORY STRIPES,” and    -   3. Utility application Ser. No. 12/______ filed on even date        herewith, and entitled “DISTRIBUTED STORAGE NETWORK MEMORY        ACCESS BASED ON MEMORY STATE,”        all of which are incorporated herein for all purposes.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not Applicable

INCORPORATION-BY-REFERENCE OF MATERIAL SUBMITTED ON A COMPACT DISC

Not Applicable

BACKGROUND OF THE INVENTION

1. Technical Field of the Invention

This invention relates generally to computing and more particularly tostorage of information.

2. Description of Related Art

Computing systems are known to communicate, process, and store data.Such computing systems range from wireless smart phones to data centersthat support millions of web searches, stock trades, or on-linepurchases every day. Computing processing is known to manipulate datafrom one form into another. For instance, raw picture data from an imagesensor may be compressed, or manipulated, in accordance with a picturecompression standard to produce a standardized compressed picture thatcan be saved or shared with others. Computer processing capabilitycontinues to advance as processing speed advances and softwareapplications that perform the manipulation become more sophisticated.

With the advances in computing processing speed and communication speed,computers manipulate real time media from voice to streaming highdefinition video. Purpose-built communications devices, like the phone,are being replaced by more general-purpose information appliances. Forexample, smart phones can support telephony communications but they arealso capable of text messaging, and accessing the internet to performfunctions including email, web browsing, remote applications access, andmedia communications. Media communications includes telephony voice,image transfer, music files, video files, real time video streaming andmore.

Each type of computing system is constructed, and hence operates, inaccordance with one or more communication, processing, and storagestandards. With such standards, and with advances in technology, moreand more of the global information content is being converted intoelectronic formats. For example, more digital cameras are now being soldthan film cameras, thus producing more digital pictures. High growthrates exist for web based programming that until recently was allbroadcast by just a few over the air television stations and cabletelevision providers. Digital content standards, such as used inpictures, papers, books, video entertainment, home video, all enablethis global transformation to a digital format. Electronic contentpervasiveness is producing increasing demands on the storage function ofcomputing systems.

A typical computer storage function includes one or more memory devicesto match the needs of the various operational aspects of the processingand communication functions. For example, a memory device may includesolid-state NAND flash, random access memory (RAM), read only memory(ROM), a mechanical hard disk drive. Each type of memory device has aparticular performance range and normalized cost. The computing systemarchitecture optimizes the use of one or more types of memory devices toachieve the desired functional and performance goals of the computingsystem. Generally, the immediacy of access dictates what type of memorydevice is used. For example, RAM memory can be accessed in any randomorder with a constant response time. By contrast, memory devicetechnologies that require physical movement such as magnetic discs,tapes, and optical discs, have a variable responses time as the physicalmovement can take longer than the data transfer.

Each type of computer storage system is constructed, and hence operates,in accordance with one or more storage standards. For instance, computerstorage systems may operate in accordance with one or more standardsincluding, but not limited to network file system (NFS), flash filesystem (FFS), disk file system (DFS), small computer system interface(SCSI), internet small computer system interface (iSCSI), file transferprotocol (FTP), and web-based distributed authoring and versioning(WebDAV). An operating systems (OS) and storage standard may specify thedata storage format and interface between the processing subsystem andthe memory devices. The interface may specify a structure such asdirectories and files. Typically a memory controller provides aninterface function between the processing function and memory devices.As new storage systems are developed, the memory controller functionalrequirements may change to adapt to new standards.

Memory devices may fail, especially those that utilize technologies thatrequire physical movement like a disc drive. For example, it is notuncommon for a disc drive to suffer from bit level corruption on aregular basis, or complete drive failure after an average of three yearsof use. One common solution is to utilize more costly disc drives thathave higher quality internal components. Another solution is to utilizemultiple levels of redundant disc drives to abate these issues byreplicating the data into two or more copies. One such redundant driveapproach is called redundant array of independent discs (RAID). Multiplephysical discs comprise an array where parity data is added to theoriginal data before storing across the array. The parity is calculatedsuch that the failure of one or more discs will not result in the lossof the original data. The original data can be reconstructed from theother discs. RAID 5 uses three or more discs to protect data from thefailure of any one disc. The parity and redundancy overhead reduces thecapacity of what three independent discs can store by one third(n-1=3-2=2 discs of capacity using 3 discs). RAID 6 can recover from aloss of two discs and requires a minimum of four discs with anefficiency of n-2. Typical RAID systems utilize a RAID control to encodeand decode the data across the array.

Drawbacks of the RAID approach include effectiveness, efficiency andsecurity. As more discs are added, the probability of one or two discsfailing rises and is not negligible, especially if more desired lesscostly discs are used. When one disc fails, it should be immediatelyreplaced and the data reconstructed before a second drive fails. Toprovide high reliability over a long time period, and if the RAID arrayis part of a national level computing system with occasional siteoutages, it is also common to mirror RAID arrays at different physicallocations. Unauthorized file access becomes a more acute problem whenwhole copies of the same file are replicated, either on just one storagesystem site or at two or more sites. In light of the effectiveness, theefficiency of dedicating 1 to 2 discs per array for the RAID overhead isan issue.

Therefore, a need exists to provide a data storage solution thatprovides more effective timeless continuity of data, minimizes adverseaffects of multiple memory elements failures, provides improvedsecurity, can be adapted to a wide variety storage system standards andis compatible with computing and communications systems.

BRIEF SUMMARY OF THE INVENTION

The present invention is directed to apparatus and methods of operationthat are further described in the following Brief Description of theDrawings, the Detailed Description of the Invention, and the claims.Various features and advantages of the present invention will becomeapparent from the following detailed description of the invention madewith reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)

FIG. 1 is a schematic block diagram of an embodiment of a computingsystem in accordance with the invention;

FIG. 2 is a schematic block diagram of an embodiment of a computing corein accordance with the invention;

FIG. 3 is a schematic block diagram of an embodiment of a distributedstorage processing unit in accordance with the invention;

FIG. 4 is a schematic block diagram of an embodiment of a distributedstorage unit in accordance with the invention;

FIG. 5 is a flowchart illustrating the reading and writing of memory;

FIG. 6 is a state transition diagram illustrating the reading andwriting of memory;

FIG. 7 is a flowchart illustrating the writing of memory;

FIG. 8A is a schematic block diagram of an embodiment of a distributedstorage system in accordance with the invention;

FIG. 8B is another flowchart illustrating the writing of memory;

FIG. 9A is a schematic block diagram of another embodiment of adistributed storage system in accordance with the invention;

FIG. 9B is another flowchart illustrating the writing of memory;

FIG. 10 is a schematic block diagram of another embodiment of adistributed storage system in accordance with the invention; and

FIG. 11 is another flowchart illustrating the writing of memory.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 is a schematic block diagram of a computing system 10 thatincludes one or more of a first type of user devices 12, one or more ofa second type of user devices 14, at least one distributed storage (DS)processing unit 16, at least one DS managing unit 18, at least onestorage integrity processing unit 20, and a distributed storage network(DSN) memory 22 coupled via a network 24. The network 24 may include oneor more wireless and/or wire lined communication systems; one or moreprivate intranet systems and/or public internet systems; and/or one ormore local area networks (LAN) and/or wide area networks (WAN).

The DSN memory 22 includes a plurality of distributed storage (DS) units36 for storing data of the system. Each of the DS units 36 includes aprocessing module and memory and may be located at a geographicallydifferent site than the other DS units (e.g., one in Chicago, one inMilwaukee, etc.). The processing module may be a single processingdevice or a plurality of processing devices. Such a processing devicemay be a microprocessor, micro-controller, digital signal processor,microcomputer, central processing unit, field programmable gate array,programmable logic device, state machine, logic circuitry, analogcircuitry, digital circuitry, and/or any device that manipulates signals(analog and/or digital) based on hard coding of the circuitry and/oroperational instructions. The processing module may have an associatedmemory and/or memory element, which may be a single memory device, aplurality of memory devices, and/or embedded circuitry of the processingmodule. Such a memory device may be a read-only memory, random accessmemory, volatile memory, non-volatile memory, static memory, dynamicmemory, flash memory, cache memory, and/or any device that storesdigital information. Note that if the processing module includes morethan one processing device, the processing devices may be centrallylocated (e.g., directly coupled together via a wired and/or wireless busstructure) or may be distributedly located (e.g., cloud computing viaindirect coupling via a local area network and/or a wide area network).Further note that when the processing module implements one or more ofits functions via a state machine, analog circuitry, digital circuitry,and/or logic circuitry, the memory and/or memory element storing thecorresponding operational instructions may be embedded within, orexternal to, the circuitry comprising the state machine, analogcircuitry, digital circuitry, and/or logic circuitry. Still further notethat, the memory element stores, and the processing module executes,hard coded and/or operational instructions corresponding to at leastsome of the steps and/or functions illustrated in FIGS. 1-11.

Each of the user devices 12-14, the DS processing unit 16, the DSmanaging unit 18, and the storage integrity processing unit 20 may be aportable computing device (e.g., a social networking device, a gamingdevice, a cell phone, a smart phone, a personal digital assistant, adigital music player, a digital video player, a laptop computer, ahandheld computer, a video game controller, and/or any other portabledevice that includes a computing core) and/or a fixed computing device(e.g., a personal computer, a computer server, a cable set-top box, asatellite receiver, a television set, a printer, a fax machine, homeentertainment equipment, a video game console, and/or any type of homeor office computing equipment). Such a portable or fixed computingdevice includes a computing core 26 and one or more interfaces 30, 32,and/or 33. An embodiment of the computing core 26 will be described withreference to FIG. 2.

With respect to the interfaces, each of the interfaces 30, 32, and 33includes software and/or hardware to support one or more communicationlinks via the network 24 and/or directly. For example, interfaces 30support a communication link (wired, wireless, direct, via a LAN, viathe network 24, etc.) between the first type of user device 14 and theDS processing unit 16. As another example, DSN interface 32 supports aplurality of communication links via the network 24 between the DSNmemory 22 and the DS processing unit 16, the first type of user device12, and/or the storage integrity processing unit 20. As yet anotherexample, interface 33 supports a communication link between the DSmanaging unit 18 and any one of the other devices and/or units 12, 14,16, 20, and/or 22 via the network 24.

In general, the system 10 supports three primary functions: distributednetwork data storage management, distributed data storage and retrieval,and data storage integrity verification. In accordance with these threeprimary functions, data can be distributedly stored in a plurality ofphysically different locations and subsequently retrieved in a reliableand secure manner regardless of failures of individual storage devices,failures of network equipment, the duration of storage, the amount ofdata being stored, attempts at hacking the data, etc.

The DS managing unit 18 performs the distributed network data storagemanagement functions, which include establishing distributed datastorage parameters, performing network operations, performing networkadministration, and/or performing network maintenance. The DS managingunit 18 establishes the distributed data storage parameters (e.g.,allocation of virtual DSN memory space, distributed storage parameters,security parameters, billing information, user profile information,etc.) for one or more of the user devices 12-14 (e.g., established forindividual devices, established for a user group of devices, establishedfor public access by the user devices, etc.). For example, the DSmanaging unit 18 coordinates the creation of a vault (e.g., a virtualmemory block) within the DSN memory 22 for a user device (for a group ofdevices, or for public access). The DS managing unit 18 also determinesthe distributed data storage parameters for the vault. In particular,the DS managing unit 18 determines a number of slices (e.g., the numberthat a data segment of a data file and/or data block is partitioned intofor distributed storage) and a threshold value (e.g., the minimum numberof slices required to reconstruct the data segment).

As another example, the DS managing module 18 may create and storelocally or within the DSN memory 22 user profile information. The userprofile information includes one or more of authentication information,permissions, and/or the security parameters. The security parameters mayinclude one or more of encryption/decryption scheme, one or moreencryption keys, key generation scheme, and data encoding/decodingscheme.

As yet another example, the DS managing unit 18 may create billinginformation for a particular user, user group, vault access, publicvault access, etc. For instance, the DS managing unit 18 may track thenumber of times user accesses a private vault and/or public vaults,which can be used to generate a per-access bill. In another instance,the DS managing unit 18 tracks the amount of data stored and/orretrieved by a user device and/or a user group, which can be used togenerate a per-data-amount bill.

The DS managing unit 18 also performs network operations, networkadministration, and/or network maintenance. As at least part ofperforming the network operations and/or administration, the DS managingunit 18 monitors performance of the devices and/or units of the system10 for potential failures, determines the devices and/or unit'sactivation status, determines the devices' and/or units' loading, andany other system level operation that affects the performance level ofthe system 10. For example, the DS managing unit 18 may receive andaggregate network management alarms, alerts, errors, status information,performance information, and messages from the devices 12-14 and/or theunits 16, 20, 22. For example, the DS managing unit 18 may receive asimple network management protocol (SNMP) message regarding the statusof the DS processing unit 16.

The DS managing unit 18 performs the network maintenance by identifyingequipment within the system 10 that needs replacing, upgrading,repairing, and/or expanding. For example, the DS managing unit 18 maydetermine that the DSN memory 22 needs more DS units 36 or that one ormore of the DS units 36 needs updating.

The second primary function of distributed data storage and retrievalfunction begins and ends with a user device 12-14. For instance, if asecond type of user device 14 has a data file 38 and/or data block 40 tostore in the DSN memory 22, it send the data file 38 and/or data block40 to the DS processing unit 16 via its interface 30. As will bedescribed in greater detail with reference to FIG. 2, the interface 30functions to mimic a conventional operating system (OS) file systeminterface (e.g., network file system (NFS), flash file system (FFS),disk file system (DFS), file transfer protocol (FTP), web-baseddistributed authoring and versioning (WebDAV), etc.) and/or a blockmemory interface (e.g., small computer system interface (SCSI), internetsmall computer system interface (iSCSI), etc.). In addition, theinterface 30 may attach a user identification code (ID) to the data file38 and/or data block 40.

The DS processing unit 16 receives the data file 38 and/or data block 40via its interface 30 and performs a distributed storage (DS) process 34thereon. The DS processing 34 begins by partitioning the data file 38and/or data block 40 into one or more data segments, which isrepresented as Y data segments. For example, the DS processing 34 maypartition the data file 38 and/or data block 40 into a fixed byte sizesegment (e.g., 2¹ to 2¹¹ bytes, where n=>2) or a variable byte size(e.g., change byte size from segment to segment, or from groups ofsegments to groups of segments, etc.).

For each of the Y data segments, the DS processing 34 error encodes(e.g., forward error correction (FEC), information dispersal algorithm,or error correction coding) and slices (or slices then error encodes)the data segment into a plurality of error coded (EC) data slices 42-48,which is represented as X slices per data segment. The number of slices(X) per segment, which corresponds to a number of pillars n, is set inaccordance with the distributed data storage parameters and the errorcoding scheme. For example, if a Reed-Solomon (or other FEC scheme) isused in an n/k system, then a data segment is divided into n slices,where k number of slices is needed to reconstruct the original data(i.e., k is the threshold). As a few specific examples, the n/k factormay be 5/3; 6/4; 8/6; 8/5; 16/10.

For each slice 42-48, the DS processing unit 16 creates a unique slicename and appends it to the corresponding slice 42-48. The slice nameincludes universal DSN memory addressing routing information (e.g.,virtual memory addresses in the DSN memory 22) and user-specificinformation (e.g., user ID, file name, data block identifier, etc.).

The DS processing unit 16 transmits the plurality of EC slices 42-48 toa plurality of DS units 36 of the DSN memory 22 via the DSN interface 32and the network 24. The DSN interface 32 formats each of the slices fortransmission via the network 24. For example, the DSN interface 32 mayutilize an internet protocol (e.g., TCP/IP, etc.) to packetize theslices 42-48 for transmission via the network 24.

The number of DS units 36 receiving the slices 42-48 is dependent on thedistributed data storage parameters established by the DS managing unit18. For example, the DS managing unit 18 may indicate that each slice isto be stored in a different DS unit 36. As another example, the DSmanaging unit 18 may indicate that like slice numbers of different datasegments are to be stored in the same DS unit 36. For example, the firstslice of each of the data segments is to be stored in a first DS unit36, the second slice of each of the data segments is to be stored in asecond DS unit 36, etc. In this manner, the data is encoded anddistributedly stored at physically diverse locations to improved datastorage integrity and security. Further examples of encoding the datasegments will be provided with reference to one or more of FIGS. 2-11.

Each DS unit 36 that receives a slice 42-48 for storage translates thevirtual DSN memory address of the slice into a local physical addressfor storage. Accordingly, each DS unit 36 maintains a virtual tophysical memory mapping to assist in the storage and retrieval of data.

The first type of user device 12 performs a similar function to storedata in the DSN memory 22 with the exception that it includes the DSprocessing. As such, the device 12 encoded and slices the data fileand/or data block it has to store. The device then transmits the slices35 to the DSN memory via its DSN interface 32 and the network 24.

For a second type of user device 14 to retrieve a data file or datablock from memory, it issues a read command via its interface 30 to theDS processing unit 16. The DS processing unit 16 performs the DSprocessing 34 to identify the DS units 36 storing the slices of the datafile and/or data block based on the read command. The DS processing unit16 may also communicate with the DS managing unit 18 to verify that theuser device 14 is authorized to access the requested data.

Assuming that the user device is authorized to access the requesteddata, the DS processing unit 16 issues slice read commands to at least athreshold number of the DS units 36 storing the requested data (e.g., toat least 10 DS units for a 16/10 error coding scheme). Each of the DSunits 36 receiving the slice read command, verifies the command,accesses its virtual to physical memory mapping, retrieves the requestedslice, or slices, and transmits it to the DS processing unit 16.

Once the DS processing unit 16 has received a threshold number of slicesfor a data segment, it performs an error decoding function andde-slicing to reconstruct the data segment. When Y number of datasegments has been reconstructed, the DS processing unit 16 provides thedata file 38 and/or data block 40 to the user device 14. Note that thefirst type of user device 12 performs a similar process to retrieve adata file and/or data block.

The storage integrity processing unit 20 performs the third primaryfunction of data storage integrity verification. In general, the storageintegrity processing unit 20 periodically retrieves slices 45 of a datafile or data block of a user device to verify that one or more sliceshas not been corrupted or lost (e.g., the DS unit failed). The retrievalprocess mimics the read process previously described.

If the storage integrity processing unit 20 determines that one or moreslices is corrupted or lost, it rebuilds the corrupted or lost slice(s)in accordance with the error coding scheme. The storage integrityprocessing unit 20 stores the rebuild slice, or slices, in theappropriate DS unit(s) 36 in a manner that mimics the write processpreviously described.

FIG. 2 is a schematic block diagram of an embodiment of a computing core26 that includes a processing module 50, a memory controller 52, mainmemory 54, a video graphics processing unit 55, an input/output (IO)controller 56, a peripheral component interconnect (PCI) interface 58,at least one IO device interface module 62, a read only memory (ROM)basic input output system (BIOS) 64, and one or more memory interfacemodules. The memory interface module(s) includes one or more of auniversal serial bus (USB) interface module 66, a host bus adapter (HBA)interface module 68, a network interface module 70, a flash interfacemodule 72, a hard drive interface module 74, and a DSN interface module76. Note the DSN interface module 76 and/or the network interface module70 may function as the interface 30 of the user device 14 of FIG. 1.Further note that the IO device interface module 62 and/or the memoryinterface modules may be collectively or individually referred to as IOports.

The processing module 50 may be a single processing device or aplurality of processing devices. Such a processing device may be amicroprocessor, micro-controller, digital signal processor,microcomputer, central processing unit, field programmable gate array,programmable logic device, state machine, logic circuitry, analogcircuitry, digital circuitry, and/or any device that manipulates signals(analog and/or digital) based on hard coding of the circuitry and/oroperational instructions. The processing module may have an associatedmemory and/or memory element, which may be a single memory device, aplurality of memory devices, and/or embedded circuitry of the processingmodule. Such a memory device may be a read-only memory, random accessmemory, volatile memory, non-volatile memory, static memory, dynamicmemory, flash memory, cache memory, and/or any device that storesdigital information. Note that if the processing module includes morethan one processing device, the processing devices may be centrallylocated (e.g., directly coupled together via a wired and/or wireless busstructure) or may be distributedly located (e.g., cloud computing viaindirect coupling via a local area network and/or a wide area network).Further note that when the processing module implements one or more ofits functions via a state machine, analog circuitry, digital circuitry,and/or logic circuitry, the memory and/or memory element storing thecorresponding operational instructions may be embedded within, orexternal to, the circuitry comprising the state machine, analogcircuitry, digital circuitry, and/or logic circuitry. Still further notethat, the memory element stores, and the processing module executes,hard coded and/or operational instructions corresponding to at leastsome of the steps and/or functions illustrated in FIGS. 1-11.

FIG. 3 is a schematic block diagram of an embodiment of a dispersedstorage (DS) processing unit 16 and/or of the DS processing module 34 ofuser device 12 (see FIG. 1). The DS processing unit 16 includes agateway module 107, an access module 109, a grid module 84, a storagemodule 113, and a bypass/feedback path. The DS processing unit 16 mayalso include an interface 30 and the DSnet interface 32.

In an example of storing data, the gateway module 107 of the DSprocessing unit 16 receives an incoming data object (e.g., a data file,a data block, an EC data slice, etc.), authenticates the user associatedwith the data object, obtains user information of the authenticateduser, and assigns a source name to the data object in accordance withthe user information. To authenticate the user, the gateway module 107verifies the user ID 119 with the managing unit 18 (see FIG. 1) and/oranother authenticating unit. If the user ID is verified, the gatewaymodule 107 retrieves the user information from the managing unit 18 (seeFIG. 1), the user device 14, and/or the other authenticating unit basedon the user ID.

The user information includes a vault identifier, operationalparameters, and user attributes (e.g., user data, billing information,etc.). A vault identifier identifies a vault, which is a virtual memoryspace that maps to a set of DS storage units 36. For example, vault 1(i.e., user 1's DSN memory space) includes eight DS storage units (X=8wide) and vault 2 (i.e., user 2's DSN memory space) includes sixteen DSstorage units (X=16 wide). The operational parameters may include anerror coding algorithm, the width n (number of pillars X or slices persegment for this vault), a read threshold T, an encryption algorithm, aslicing parameter, a compression algorithm, an integrity check method,caching settings, parallelism settings, and/or other parameters that maybe used to access the DSN memory layer.

The gateway module 107 determines the source name to associate with thedata object based on the vault identifier and the data object. Forexample, the source name may contain a data name (block number or a filenumber), the vault generation number, a reserved field, and a vaultidentifier. The data name may be randomly assigned but is associatedwith the user data object.

The gateway module 107 may utilize the bypass/feedback path to transferan incoming EC data slice to another DS storage unit 36 (see FIG. 1)when the DS processing module 34 determines that the EC data should betransferred. Alternatively, or in addition to, the gateway module 60 mayuse the bypass/feedback path to feedback an EC slice for sub-slicing.

The access module 109 receives the data object and creates a series ofdata segments 1 through Y therefrom. The number of segments Y may bechosen or random based on a selected segment size and the size of thedata object. For example, if the number of segments is chosen to be afixed number, then the size of the segments varies as a function of thesize of the data object. For instance, if the data object is an imagefile of 4,194,304 eight bit bytes (e.g., 33,554,432 bits) and the numberof segments Y=131,072, then each segment is 256 bits or 32 bytes. Asanother example, if segment sized is fixed, then the number of segmentsY varies based on the size of data object. For instance, if the dataobject is an image file of 4,194,304 bytes and the fixed size of eachsegment is 4,096 bytes, the then number of segments Y=1,024. Note thateach segment is associated with the source name.

The grid module 84, as previously discussed, may pre-manipulate (e.g.,compression, encryption, cyclic redundancy check (CRC), etc.) the datasegment before creating X error coded data slices for each data segment.The grid module 84 creates XY error coded data slices for the Y datasegments of the data object. The grid module 84 adds forward errorcorrection bits to the data segment bits in accordance with an errorcoding algorithm (e.g., Reed-Solomon, Convolution encoding, Trellisencoding, etc.) to produce an encoded data segment. The grid module 84determines the slice name and attaches the unique slice name to each ECdata slice.

The number of pillars, or slices X per data segment (e.g., X=16) ischosen as a function of the error coding objectives. The DS processingmodule may utilize different error coding parameters for EC data slicesand EC data sub-slices based on guidance from one or more of a uservault (e.g., stored parameters for this user), a command from the DSmanaging unit or other system element, priority of the EC data slice,type of data in the EC data slice, and/or retrieval speed requirements.A read threshold T (e.g., T=10) of the error coding algorithm is theminimum number of error-free error coded data slices required to be ableto reconstruct a data segment. The DS processing unit can compensate forX−T (e.g., 16−10=6) missing, out-of-date, and/or corrupted error codeddata slices per data segment.

The grid module 84 receives each data segment 1-Y and, for each datasegment generates X number of error coded (EC) slices using an errorcoding function. The grid module 84 also determines the DS storage units36 for storing the EC data slices based on a dispersed storage memorymapping associated with the user's vault and/or DS storage unit 36attributes, which include availability, self-selection, performancehistory, link speed, link latency, ownership, available DSN memory,domain, cost, a prioritization scheme, a centralized selection messagefrom another source, a lookup table, data ownership, and/or any otherfactor to optimize the operation of the computing system.

The storage module 113 may perform integrity checks on the EC dataslices and then transmit the EC data slices 1 through X of each segment1 through Y to the DS storage units. The DS storage units 36 may storethe EC data slices and locally keep a table to convert virtual DSNaddresses into physical storage addresses. Note that the number of DSstorage units 36 is equal to or greater than the number of pillars(slices X per segment) so that no more than one error coded data sliceof the same data segment is stored on the same DS storage unit 36.Further note that EC data slices of the same pillar number but ofdifferent segments (e.g., EC data slice 1 of data segment 1 and EC dataslice 1 of data segment 2) may be stored on the same or different DSstorage units 36 (see FIG. 1).

In an example of a read operation, the user device 10 or 12 sends a readrequest to the DS processing unit 14, which authenticates the request.When the request is authentic, the DS processing unit 14 sends a readmessage to each of the DS storage units 36 storing slices of the dataobject being read. The slices are received via the DSnet interface 34and processed by the storage module 113, which performs a parity checkand provides the slices to the grid module 84. The grid module 84de-slices and decodes the slices of a data segment to reconstruct thedata segment. The access module reconstructs the data object from thedata segments and the gateway module 107 formats the data object fortransmission to the user device.

FIG. 4 is a schematic block diagram of an embodiment of a distributedstorage unit 36 that includes a storage unit control module 402, aplurality of memories 403, 404, 405, and 406, a plurality of paritymemories 408 and 409, and a cache memory 415. In another embodiment,there may be 8, 16, or more memories.

The storage unit control module 402 may be implemented with thecomputing core of FIG. 2. The memories 403-406 may be one or more of amagnetic hard disk, NAND flash, read only memory, optical disk, and/orany other type of read-only, or read/write memory. The memories may beimplemented as part of or outside of the DS storage unit. For example,memory 1 may be implemented in the DS unit and memory 4 may beimplemented in a remote server (e.g., a different DS unit operablycoupled to the DS unit via the network). In an example, memories 403-406and parity memories 408-409 are implemented with the magnetic hard disktechnology and the cache memory 415 is implemented with the NAND flashtechnology.

In some embodiments, a DS unit includes cache memory 415 implementedusing a single solid state drive (SSD). In other embodiments, all of thememories are implemented using the same type of device, and one or moreof the memories is temporarily selected for use as “cache memory” forpurposes of temporarily storing data to be written. The temporarilyselected memory can serve as a cache memory until the DS unit shiftsresponsibility for caching writes to another memory.

The storage unit control module 402 includes the DSnet interface 32 anda processing module. The storage unit control module 402 may be operablycoupled to the computing system via the DSnet interface 32 via thenetwork. The storage unit control module 402 may receive EC data slicesto store via the DSnet interface 32. In an embodiment, the storage unitcontrol module 402 determines where (e.g., which address on which of thememories) to store the received EC data slice. The determination may bebased on one or more of the metadata, a command (e.g., from the DSprocessing unit indicating which memory type to use), a type of dataindicator, a priority indicator, a memory state indicator, availablememory, memory performance data, memory cost data, the memorycharacteristics, and/or any other parameter to facilitate desired levelsof efficiency and performance. The memory state may indicate whether thememory is in a write only state, a read only state, a write with readpriority state, or some other state to indicate the availability.

The storage unit control module 402 creates and maintains a localvirtual DSN address to physical memory table. The storage unit controlmodule 402 determines where previously stored EC data slices are locatedbased on the local virtual DSN address to physical memory table uponreceiving a retrieve command via the network. The storage unit controlmodule 402 may save activity records (e.g., memory utilization, errors,stores, retrievals, etc.) as logs in any of the memories.

The storage unit control module 402 may utilize the parity memories408-409 to store and retrieve parity across the data stored in memories403-406. The storage unit control module 402 may immediately recreate aslice that is stored in a memory in the write only state based onreading the other memories in the read only state, reading the paritymemory 1 and/or parity memory 2, and calculating the desired slice. Thestorage unit control module 402 may temporarily pair a write only statememory 403-406 with a write only state parity memory 408-409 to enablerapid writes of new slices (e.g., write a slice to memory 1 and writethe parity to parity memory 1), while another parity memory in the readonly state may be available to provide the needed parity to reconstructslices that are stored on the write only state memory.

In an example, the storage unit control module 402 may choose memory 1(e.g., a magnetic hard disk drive) to store the received EC data slicesince memory 1 is in a write only state (e.g., available immediately),the memories 2-4 are in the read only state, parity memory 1 is pairedwith memory 1 in the write only state, parity memory 2 is in the readyonly state, and the memory 1 memory characteristics favorably match themetadata of the EC data slice, including performance, efficiency, cost,and response time. The storage unit control module 402 queues a readrequest in the cache memory when the requested slice is in the memory 1(but in the write state). The storage unit control module 402 mayprocess the queued read request for memory 1 by retrieving the requestfrom the cache memory, reading the memories 2-4 (e.g., the same memorystripe or common address range across each), reading the party memory 2,and calculating the desired slice.

Note that the storage unit control module 402 may queue write requestsand slices when a desired memory 403-406 is in the read only state. Thestorage unit control module may subsequently change the state of memory1 from write only to the read only state, or the write with readpriority state to enable processing of the queued read request. Notethat the DS unit 36 can immediately retrieve slices where the slices arestored in memories in the read only state, or in the write with readpriority state (e.g., memories 2-4). Further note that the DS unit 36may rotate the write only state amongst the memories 1-4 and the paritymemories 1-2 from time to time to even out the cumulative storage andoptimize performance. A method to choose the memories and change thememory state will be discussed in greater detail with reference to FIGS.5-11.

FIG. 5 is a flowchart illustrating a method 500 of reading and writingto memory where the DS unit 36 (see FIG. 4) may control the DS unitmemory state and memory utilization to optimize the performance of thememory.

The method begins where the storage unit control module 402 (see FIG. 4)checks for a received request. As illustrated by block 505, the DS unitmay receive the request from one or more of the DS processing unit 16,the user device 12, the storage integrity processing unit 20, and/or theDS managing unit 18 (see FIG. 1). As illustrated by block 507, thestorage unit control module determines the request type based on therequest when the request is received. The method branches to block 532,which illustrates receiving a slice to store when the storage unitcontrol module determines the request type is a write request.

As illustrated by block 509, the storage unit control module determinesthe slice location and state when the request type is a read request. Asillustrated by block 511, the determination is based in part onaccessing the local virtual DSN address to physical location table toidentify the memory, the address, and the memory state. As illustratedby block 513, the storage unit control module retrieves the slice basedon the memory and address when the memory state is the read state. Thestorage unit control module sends the slice to the requester and themethod branches back to look for more requests.

As illustrated by block 515, the storage unit control module determinesthe method to read the slice when the memory state is the write state.Note that in this state the memory is only writing at this time tooptimize the throughput performance of the memory requiring therequested slice to be obtained in another way other than reading itdirectly from the memory where the slice was initially stored (e.g.,which may disrupt the write state performance when the memory is a harddisk drive). As illustrated by block 519, the determination of themethod to read the slice is based on one or more of a predetermination,a command, a DS unit status indicator, a loading indicator for thememories in the read state, a priority indicator, and/or any otherindicator to optimize the memory performance. As illustrated by block517, the storage unit control module may send a read request responsemessage to the requester where the response denies the request when thestorage unit control module determines the method to be to utilizeanother DS unit. Note that in this scenario the DS unit does not returnthe requested slice to the requester but instead informs the requesterthat no slice will be returned. The requester must rely onreconstructing the original data object based on the retrieving theslices from the other pillars and performing the de-slicing and decodingsteps. In another embodiment, the requester may repeat the read requestto the DS unit with a priority indicator set when the process toreconstruct the data object fails since a read threshold of k goodslices are not retrieved from the DS units.

In various embodiments, including embodiments in which a DS unit uses anSSD cache or where responsibility for caching writes is delegated tovarious different memories within a DS unit, the DS unit always respondsto read requests, and implementation of block 517 is not required.

As illustrated by block 521, the storage unit control module mayreconstruct the slice from a reverse parity operation based on reading aportion of the memories (e.g., a logical stripe across the memories) andparity memory in the read state when the storage unit control moduledetermines the method to be to utilize the DS unit now. As illustratedby block 523, the storage unit control module sends the slice to therequester and returns to the step to look for received requests.

Handling the write request begins, as illustrated by block 532, with thestorage unit control module receiving the slice to store in the writerequest. As illustrated by block 534, the storage unit control moduledetermines the present write state memory based on the local virtual DSNaddress to physical address table. As illustrated by block 536, thestorage unit control module stores the slice in the write state memoryand updates the write parity memory by reading a corresponding portionof the read state memories (e.g., same logical stripe across thememories) and calculating the parity across the slice just written tothe write state memory and the read state memories. The storage unitcontrol module stores the parity to the write state parity memory, asshown by block 538.

As illustrated by block 540, the storage unit control module determinesif it is time to rotate the write state memory and write state paritymemory to different memories. The determination may be based on one ormore of a timer expiration since the last rotation, a command, a memoryutilization indicator (e.g., the present write state memory is fillingup), a read request history indicator (e.g., many read requests forslices in the write state memory), and/or any other indicator tooptimize the memory performance. As illustrated by block 542, the methodbranches back to look for received requests when the storage unitcontrol module determines it is not time to rotate the write statememory.

As illustrated by block 544, the storage unit control module determinesthe next write state memory and write state parity memory when thestorage unit control module determines it is time to rotate the writestate memory. The determination may be based on one or more ofidentifying which memory was in the write state least recently, apredetermination, a rotation order indicator, a command, a memoryutilization indicator (e.g., choose a memory with the most availableunused space), a read request history indicator (e.g., avoid a memorywith a higher read request frequency than other memories), and/or anyother indicator to optimize the memory performance. The storage unitcontrol module updates the local virtual DSN address to physicallocation table with the chosen write state memory and write state paritymemory. As illustrated by block 546, the storage unit control moduleupdates the local virtual DSN address to physical location table tomodify the state of the previous write state memory and write stateparity memory from write state to the read state. Additionally, slicescan be moved back to their proper drives. The method branches back tolook for received requests.

In another embodiment, the number of write state memories may be two ormore to further improve the write performance of the DS unit. Thestorage unit control module may only rotate one memory at a time fromthe write state to the read state or the storage unit control module mayrotate more than one memory at a time from the write state to the readstate.

FIG. 6 is a state transition diagram 600 illustrating the reading andwriting of memory where the DS unit may control the DS unit memory state601 and memory utilization to optimize the performance of the memory.There are three states of the memory: the read only state 607, the writeonly state 603, and the write state with read priority605.

The storage unit control module determines the memory state andprocesses received read and write requests based on the memory state tooptimize the memory performance. For example, when the memory is in theread only state 607, the storage unit control module processes only readrequests, unless too many write requests are pending (e.g., the numberwrite requests is greater than a high threshold). In another example,when the memory is in the write only state 603, the storage unit controlmodule processes only write requests until the pending write requestsare reduced to a low threshold level. In another example, when thememory is in the write state with read priority 605, the storage unitcontrol module opportunistically processes any pending write requestsunless there are pending read requests.

In various embodiments, including embodiments in which a DS unit uses anSSD cache or where responsibility for caching writes is delegated tovarious different memories within a DS unit, the DS unit always respondsto read requests. In such embodiments, a particular piece of memorybeing in write only mode 603 means that a read will be delayed, and datawill always be stored immediately in read cache memory.

Note that in all memory states 601, the storage unit control modulequeues received read requests into a read queue and received writerequests into a write queue by storing the request (and slice in thecase of a write request) in the cache memory as indicated by the upperright portion of FIG. 6. The requests may be subsequently de-queued andprocessed as discussed below.

Starting with the read only state, the storage unit control moduledetermines if the read queue is not empty and de-queues the readrequest, determines the memory location, retrieves the slice, and sendsthe slice to the requester when the storage unit control moduledetermines the read queue is not empty. The storage unit control moduledetermines if the write queue is above the high threshold of writerequests while the memory is in the read only state. The storage unitcontrol module changes the state of the memory from the read only stateto the write only state when the storage unit control module determinesthat the write queue is above the high threshold of write requests. Thestorage unit control module determines if the read queue is empty whilethe memory is in the read only state. The storage unit control modulechanges the state of the memory from the read only state to the writestate with read priority when the storage unit control module determinesthat the read queue is empty.

While in the write only state (e.g., the second state of three states)the storage unit control module determines if the write queue is notempty and de-queues the write request with slice from the cache memory,determines the memory location, stores the slice, and updates the localvirtual DSN address to physical storage table when the storage unitcontrol module determines the write queue is not empty. The storage unitcontrol module determines if the write queue is below the low thresholdof write requests while the memory is in the write only state. Thestorage unit control module changes the state of the memory from thewrite only state to the read only state when the storage unit controlmodule determines that the write queue is below the low threshold ofwrite requests.

While in the write state with read priority (e.g., the third state ofthree states) the storage unit control module determines if the writequeue is not empty and de-queues the write request with slice from thecache memory, determines the memory location, stores the slice, andupdates the local virtual DSN address to physical storage table when thestorage unit control module determines the write queue is not empty. Thestorage unit control module determines if the read queue is not emptywhile the memory is in the write state with read priority. The storageunit control module changes the state of the memory from the write statewith read priority to the read only state when the storage unit controlmodule determines that the read queue is not empty.

FIG. 7 is a flowchart illustrating a method 700 of writing memory wherethe DS processing unit (or DS unit) may employ a memory diversity schemeto choose memories to store slices such that the overall systemreliability is improved. For example, the memory diversity scheme mayensure that a read threshold of k slices are stored in pillar memoriesthat are each of a different model to avoid unrecoverable data due to apotentially common memory design defect.

As illustrated by block 701, the DS processing unit creates the slicesfor distributed storage. As illustrated by block 703, the DS processingunit determines the slice metadata based on one or more of a file type,file size, priority, a security index, estimated storage time, estimatedtime between retrievals and more. As illustrated by block 705, the DSprocessing unit determines the similarity requirements and differencerequirements, sometimes referred to as diversity preferences, based onthe metadata. Similarity requirements drive similar attributes of thepillar memory choices and difference requirements drive differenceattributes of the pillar memory choices. For example, a preference orrequirement for a relatively short estimated time between retrievals maydrive pillar memory choices that all share a similar fast retrievalcharacteristic to speed frequent retrievals. Other examples ofsimilarity preferences and requirements may include similar cost andsimilar capacity. In another example, a preference or requirement forvery high reliability may drive pillar memory choices that all have adifferent memory model to improve the reliability of retrievals. Otherexamples of difference requirements and preferences may includedifferent operating systems and different installation sites.

As illustrated by block 709, the DS processing unit determines the DSunit memory characteristics for one or more candidate DS units. Thedetermination may be via a table lookup or a real time request to eachDS unit to query for the memory characteristics. The memorycharacteristics may include one or more of memory model, memory type,total capacity, available capacity, access speed, error history,estimated mean time between failures, actual mean time between failures,and/or hours of operation.

As illustrated by block 711, the DS processing unit sorts the DS unitsthat favorably match the similarity requirements and differencerequirements based on comparing the requirements to the memorycharacteristics. For example, DS units with memory that has a fastaccess memory characteristic may be sorted to favorably match the fastaccess similarity requirement. In another example, DS units with memorythat has a different model memory characteristic may be sorted tofavorably match the reliability-driven different-model requirement orpreference.

As illustrated by block 713, the DS processing unit determines the bestmatch of DS unit memories to the diversity preferences or requirementsbased on the sort if possible, or at least a favorable match. Forexample, the DS processing unit may choose at most n-k DS unit memorieswith the same model, similar error histories, or similar total hours toimprove the reliability of data object retrieval. In other words, the DSunit may choose the read threshold k of DS unit memories that has themost different models, error histories, and total hours as the memorydiversity scheme.

As illustrated by block 715, the DS processing unit sends the slices tothe chosen DS units with the best match of memory characteristics torequirements and updates the virtual DSN address to physical locationtable with the locations of the slices.

In at least some embodiments where a DS unit includes multiple memorydevices, the DS unit may implement similar functionality to thatdiscussed above to select available memory units that favorably matchthe diversity preferences determined from the slice metadata.

FIG. 8A is a schematic block diagram of an embodiment of a distributedstorage system that includes the DS processing unit 16, a temporarymemory 802, and a plurality of DS units 36. Consider an example in whichDS unit 4 may not be available due to a site outage, a DS unit failure,and/or the network is not available at DS unit 4 site. The DS processingunit 16 may temporarily store new pillar 4 slices in the temporarymemory, and/or yet another DS unit, for subsequent storage in DS unit 4.As used herein, the term “cache memory” refers to a memory that can beused temporarily store information and includes but is not limited to,cache memories such as those included in various processorarchitectures, memory specifically designated as cache memory, and thelike. The term “cache memory” is also used in a less rigorous sense torefer to any type of memories used for substantially non-permanentinformation storage. The method of operation to determine where totemporarily store the slices will be discussed in greater detail withreference to FIGS. 8B and 9B.

FIG. 8B is another flowchart illustrating a method 800 of writing tomemory where the DS processing unit 16 determines where to store newlycreated slices when at least one primary DS unit 36 is not available.

The method 800 begins as illustrated by block 803, where the DSprocessing unit creates the n slices for each data segment for storage.As illustrated by block 805, the DS processing unit determines thedesired primary DS units in which to store the slices based in part on apredetermination of the slice name in the user vault, or in the virtualDSN address to physical location table.

As illustrated by block 807, the DS processing unit determines thestatus of the chosen primary DS units based on one or more of a statustable lookup and/or a real time query to the DS unit. For example, thestatus indicates not available if the network is down to the DS unit, orif the DS unit is down. As illustrated by block 810, the DS processingunit determines the number of primary DS units that are in the readystatus. As illustrated by block 809, the DS processing unit tries otherDS units and returns to the step to determine which DS units when thenumber of ready primary DS units is less than the read threshold k. Notethat the threshold for this scenario may be k+1, k+2, or etc. in anotherembodiment to further improve the probability of subsequent data objectrecreation.

As illustrated by block 811, the DS processing unit sends the n slicesto the chosen primary DS units when the DS processing unit determinesthat the number of ready primary DS units is all n (e.g., all pillarsready). The method then continues to the step to create more slices.

As illustrated by block 813, the DS processing unit sends slices to theavailable chosen primary DS units when the DS processing unit determinesthat the number of ready primary DS units is greater than or equal tothe read threshold k but is less than all n. As illustrated by block815, the DS processing unit temporarily stores slices by storing slicesin temporary memory for any chosen primary DS units that are notavailable.

As illustrated by block 817, the DS processing unit determines if thestatus of any unavailable chosen primary DS units has changed to ready.As illustrated by blocks 819 and 821, the DS processing unit retrievesthe slices from temporary memory and sends the slices to the ready DSunit when the DS processing unit determines that the status of theunavailable chosen primary DS unit has changed to ready. As illustratedby block 823, the DS processing unit determines if all the temporarilycached slices have been stored in the chosen DS unit and continues tothe step of determining if the status has changed when all the cachedslices have not been stored in the chosen DS units. In anotherembodiment, a timeout may occur where the DS processing unit gives up onwaiting for the ready status to change in which case the DS processingunit may try another DS unit or just not store a pillar of slices (e.g.,deleting them from the temporary memory). The DS processing unit methodgoes back to the step of creating slices when all the cached slices havebeen stored in the chosen DS units.

In some embodiments, some or all slices stored in temporary memory maybe discarded according to a discard policy. The discard policy mayspecify that slices are to be discarded after a threshold period oftime, based on an amount of available storage, or based on reliabilityof the data. For example, a data slice may be discarded only when it isno longer possible to use the data slice, when the data slice is nolonger needed, or when the data slice is deemed unreliable. Some dataslices may be given retention preference over other data slices, so thatvery data slices associated with reliable data slices already in longterm storage may be discarded in favor of data slices that may be neededto correct unreliable data slices.

FIG. 9A is a schematic block diagram of another embodiment of adistributed storage system that includes the DS processing unit 16, theplurality of DS units 36, and a plurality of associated temporarymemories 904. In one example of operation, the DS unit 4 may not beavailable due to a site outage, a DS unit failure, and/or the network isnot available at DS unit 4 site. The DS processing unit 16 maytemporarily store new pillar 4 slices in one of the temporary memories904, and/or yet another DS unit, for subsequent storage in DS unit 4.The method of operation to determine where to temporarily store theslices will be discussed in greater detail with reference to FIG. 9B.

FIG. 9B is another flowchart illustrating a method 900 of writing tomemory where the DS processing unit determines where to store newlycreated slices when at least one primary DS unit is not available.

The method begins as illustrated by block 903, where the DS processingunit creates the n slices for each data segment for storage. Asillustrated by block 905, the DS processing unit determines the desiredprimary DS units in which to store the slices based in part on apredetermination of the slice name in the user vault, or in the virtualDSN address to physical location table.

As illustrated by block 907, the DS processing unit determines thestatus of the chosen primary DS units based on one or more of a statustable lookup and/or a real time query to the DS unit. For example, thestatus indicates not available if the network is down to the DS unit orif the DS unit is down. As illustrated by block 910, the DS processingunit determines the number of primary DS units that are in the readystatus. As illustrated by block 909, the DS processing unit tries otherDS units and returns to the step to determine which DS units when thenumber of ready primary DS units is less than the read threshold k. Notethat the threshold for this scenario may be k+1 or k+2, etc. in anotherembodiment to further improve the probability of subsequent data objectrecreation.

As illustrated by block 911, the DS processing unit sends the n slicesto the chosen primary DS units when the DS processing unit determinesthat the number of ready primary DS units is all n (e.g., all pillarsready). The method 900 then continues to create more slices, asillustrated by block 903.

As illustrated by block 913, the DS processing unit sends slices to theavailable chosen primary DS units when the DS processing unit determinesthat the number of ready primary DS units is greater than or equal tothe read threshold k but is less than all n.

As illustrated by block 915, the DS processing unit determines whichtemporary memory 1-3 to utilize to temporarily store the slices for theDS unit 4 that is not ready. The determination may be based on one ormore of an even rotation across the ready DS unit temporary memories(e.g., temporary/cache memory 1, then 2, then 3, then 1 etc.), onepillar high or low from the DS unit that is not ready, a list, acommand, and/or the performance of the temporary memory. The DSprocessing unit caches slices by storing slices in the chosen temporarymemory for any chosen primary DS units that are not available.

As illustrated by block 917, the DS processing unit determines if thestatus of any unavailable chosen primary DS units 36 has changed toready. As illustrated by blocks 919 and 921, the DS processing unitretrieves the slices from the temporary memory and sends the slices tothe ready DS unit when the DS processing unit determines that the statusof the unavailable chosen primary DS unit has changed to ready. Asillustrated by block 923, the DS processing unit determines if all thetemporarily cached slices have been stored in the chosen DS unit andcontinues to the step of determining if the status has changed when allthe cached slices have not been stored in the chosen DS units. Inanother embodiment, a timeout may occur where the DS processing unitgives up on waiting for the ready status to change in which case the DSprocessing unit may try another DS unit or just not store a pillar ofslices (e.g., deleting them from the temporary memory). The DSprocessing unit method goes back to the step of creating slices when allthe cached slices have been stored in the chosen DS units.

FIG. 10 is a schematic block diagram of another embodiment of adistributed storage system that includes the DS processing unit 16, anda plurality of DS units 36. The DS units 1-4 may each include a matchingnumber of memories 1-4 in some embodiments. In another embodiment, thenumber of memories per DS unit may be 8, 16 or more.

The DS units can include a matching number of memories to facilitateorganizing memories across the DS units 1-4 as storage groups or stripes1-4. The stripes 1-4 may be physical as shown or logical such that thestripe boundaries are within the memory ranges of the memories.

The DS processing unit 16 and/or the DS units determine which memoriesacross the DS units to utilize to store slices of the same data object.Note that the overall system reliability can be improved when the numberof logical stripes is minimized such that same data segment slices arecontained within the same stripe. In an embodiment (not illustrated), alogical stripe may include memory 1 of DS unit 1, memory 4 of DS unit 2,memory 2 of DS unit 3, and memory 3 of DS unit 4. This embodiment may beundesired as it can lead to lower system reliability since a memoryfailure can affect many data sets.

In another embodiment, a logical stripe may include memory 2 of DS unit1, memory 2 of DS unit 2, memory 2 of DS unit 3, and memory 2 of DS unit4. This embodiment may be more desired as it can lead to improved systemreliability, since a memory failure can affect a more limited number ofdata sets.

In general, there are n choose m possible logical stripes where m is thenumber of memories per DS unit and n is the pillar width of the vault,and “choose” refers to the combinatorial operation for determining thenumber of distinct k-combinations. The system mean time to data loss=(stripe mean time to data loss)/(number of logical stripes). Minimizingthe number of logical stripes may improve the system reliability. The DSprocessing unit and/or DS unit may determine the provisioning andutilization of the memories into logical stripes such as to minimize thenumber of logical stripes.

In an example of operation, the DS processing unit and/or DS managingunit provision memory 1 of each of DS unit 1-4 to be stripe 1, memory 2of each of DS unit 1-4 to be stripe 2, memory 3 of each of DS unit 1-4to be stripe 3, and memory 4 of each of DS unit 1-4 to be stripe 4. TheDS processing unit and/or DS unit determines to store a pillar 1 sliceof data segment A at stripe 1 of DS unit 1 (slice Al at memory 1 of DSunit 1), slice A2 at memory 1 of DS unit 2, slice A3 at memory 1 of DSunit 3, and slice A4 at memory 1 of DS unit 4. In a similar fashion theDS processing unit and/or DS unit determines to store the slices of datasegment E in stripe 1 (E1-E4), B1-B4 and F1-F4 in stripe 2, C1-C4 andG1-G4 in stripe 3, and D1-D4 and H1-H4 in stripe 4. A method ofdetermining which stripe to utilize is discussed in greater detail withreference to FIG. 11.

In some embodiments, every DS unit receives slices from a contiguous setof segments of a data source. So, as illustrated in FIG. 10, DS unit 1would receive, in order, A1, B1, C1, D1, E1, and so on. The stripingalgorithm can be used to even the load, such that no one memory has tohandle all the input/output traffic. In an embodiment illustrated byFIG. 10, if slices from segments A-D come in at once, all 4 disks maybegin storage operations, since each of the 4 memories gets something tostore.

To achieve load balancing, some embodiments apply a random-like (butdeterministic), or round-robin process to select which memory the slicewill go to based on its name. It should be a deterministic process sothat when reading, the DS unit knows which memory to access to find thesource. For example, if the store had 8 disks, it might look at the 3least significant bits of the segment's name (which would represent anynumber from 0-7 in binary). This result would determine which of the 8disks a slice would be stored in.

In other embodiments, the least significant bits of the input sourcename are not used, because they are not guaranteed to have a uniformenough distribution. In some cases, the hash of the source name is usedto create something with an even distribution, and, the leastsignificant bits of the hash are examined. Other implementations use theresult of taking the remainder when dividing the hash result by asmaller number.

FIG. 11 is another flowchart illustrating method 1100 of writing tomemory where the DS processing unit and/or DS unit determine whichstripe to utilize.

As illustrated by block 1103, the DS unit receives a slice to store fromone of the DS processing unit, the user device, the DS managing unit, orthe storage integrity processing unit. The slice is accompanied by oneor more of the command/request to store it, the slice name, the sourcename, and or the slice metadata. As illustrated by block 1105, the DSunit determines the source name either by receiving the source name orderiving it from the slice name.

As illustrated by block 1107, the DS unit calculates a reduced lengthsource name. The reduced length source name can be calculated, forexample, using a hash (e.g., CRC) function of the source name which willalways be the same number for the same source name (e.g., vault ID,vault gen, resv, and file ID). In other instances, the reduced lengthsource name can be calculated using other suitable functions, forexample, a modulo function. Generally, any reduction function that canbe used to reduce the original source name to a smaller number that canbe used to uniquely identify a particular memory can be used. In mostcases, a reduction function can be chosen to maintain a randomdistribution among the various memories of a DS unit. The randomness ofthe file ID ensures that the hash will have desired distancingproperties to spread out the slices of data objects evenly across thestripes.

As illustrated by block 1109, the DS unit determines the memory devicebased on the hash of the source name by truncating the hash to thenumber of bits required to specify the stripe range. For example, theleast two significant bits of the hash may be utilized to specify thememory number.

As illustrated by block 1113, the DS unit updates the local virtual DSNaddress to physical location table with the memory number before storingthe slice in the chosen memory, as illustrated by block 1115

In various embodiments employing a deterministic technique to find thememory device based on the hash, as discussed for example with referenceto block 1109, there a physical location table for each element is notmaintained, because the name itself is all the information needed forthe DS unit to determine the memory location. However, such a table canbe maintained for a DS processing unit to determine which DS unit keepsa particular slice. Additionally rather than using an algorithm todetermine which memory to use, an individual DS unit can furthersubdivide its namespace range so that one memory is responsible for somecontiguous range of the namespace, with that range being a subset of theDS units entire assigned range. This technique may not allow for I/Oload balancing to the same degree as other methods, since contiguoussegments for the same source would likely all fall to one or a fewmemories, rather than most or all of them.

As may be used herein, the terms “substantially” and “approximately”provides an industry-accepted tolerance for its corresponding termand/or relativity between items. Such an industry-accepted toleranceranges from less than one percent to fifty percent and corresponds to,but is not limited to, component values, integrated circuit processvariations, temperature variations, rise and fall times, and/or thermalnoise. Such relativity between items ranges from a difference of a fewpercent to magnitude differences. As may also be used herein, theterm(s) “coupled to” and/or “coupling” and/or includes direct couplingbetween items and/or indirect coupling between items via an interveningitem (e.g., an item includes, but is not limited to, a component, anelement, a circuit, and/or a module) where, for indirect coupling, theintervening item does not modify the information of a signal but mayadjust its current level, voltage level, and/or power level. As mayfurther be used herein, inferred coupling (i.e., where one element iscoupled to another element by inference) includes direct and indirectcoupling between two items in the same manner as “coupled to”. As mayeven further be used herein, the term “operable to” indicates that anitem includes one or more of power connections, input(s), output(s),etc., to perform one or more its corresponding functions and may furtherinclude inferred coupling to one or more other items. As may stillfurther be used herein, the term “associated with”, includes directand/or indirect coupling of separate items and/or one item beingembedded within another item. As may be used herein, the term “comparesfavorably”, indicates that a comparison between two or more items,signals, etc., provides a desired relationship. For example, when thedesired relationship is that signal 1 has a greater magnitude thansignal 2, a favorable comparison may be achieved when the magnitude ofsignal 1 is greater than that of signal 2 or when the magnitude ofsignal 2 is less than that of signal 1.

The present invention has also been described above with the aid ofmethod steps illustrating the performance of specified functions andrelationships thereof. The boundaries and sequence of these functionalbuilding blocks and method steps have been arbitrarily defined hereinfor convenience of description. Alternate boundaries and sequences canbe defined so long as the specified functions and relationships areappropriately performed. Any such alternate boundaries or sequences arethus within the scope and spirit of the claimed invention.

The present invention has been described above with the aid offunctional building blocks illustrating the performance of certainsignificant functions. The boundaries of these functional buildingblocks have been arbitrarily defined for convenience of description.Alternate boundaries could be defined as long as the certain significantfunctions are appropriately performed. Similarly, flow diagram blocksmay also have been arbitrarily defined herein to illustrate certainsignificant functionality. To the extent used, the flow diagram blockboundaries and sequence could have been defined otherwise and stillperform the certain significant functionality. Such alternatedefinitions of both functional building blocks and flow diagram blocksand sequences are thus within the scope and spirit of the claimedinvention. One of average skill in the art will also recognize that thefunctional building blocks, and other illustrative blocks, modules andcomponents herein, can be implemented as illustrated or by discretecomponents, application specific integrated circuits, processorsexecuting appropriate software and the like or any combination thereof.

1. A method of operating a distributed storage processing unit, themethod comprising: creating a plurality of data slices from a datasegment, the plurality of data slices to be stored in a distributedstorage network including a plurality of storage units; identifyingselected storage units of the plurality of storage units, each of theselected storage units to store particular data slices of the pluralityof data slices; determining which of the selected storage units is in aready status, wherein a ready status indicates a readiness to receivedata slices for storage; sending data slices to selected storage unitsin the ready status; and temporarily storing in a temporary memory dataslices intended for selected storage units in a non-ready status.
 2. Themethod of claim 1, further comprising: sending data slices to theselected storage units only if a number of selected storage units in theready status is at least as great as a threshold number of data slices.3. The method of claim 2, further comprising: discarding data slicesassociated with a particular selected storage unit from the temporarymemory in accordance with a discard policy.
 4. The method of claim 1,further comprising: determining that a particular selected storage unitpreviously in a non-ready status has entered a ready status; and inresponse to the determining, moving the data slices intended for theparticular selected storage unit from the temporary memory into theparticular selected storage unit.
 5. The method of claim 1, furthercomprising: determining that a particular selected storage unit hasremained in a non-ready status longer than a timeout threshold; inresponse to the determining: selecting a replacement storage unit; andmoving the data slices intended for the particular selected storage unitfrom the temporary memory into the replacement storage unit.
 6. Themethod of claim 1, further comprising: temporarily storing the dataslices intended for selected storage units in a non-ready status in atemporary memory of the distributed storage processing unit.
 7. Themethod of claim 1, further comprising: temporarily storing the dataslices intended for a particular selected storage unit in a non-readystatus in a temporary memory of the particular selected storageprocessing unit.
 8. A distributed storage processing unit comprising: aprocessor to: create a plurality of data slices from a data segment, theplurality of data slices to be stored in a distributed storage networkincluding a plurality of storage units; identify selected storage unitsof the plurality of storage units, each of the selected storage units tostore particular data slices of the plurality of data slices; determinewhich selected storage units is in a ready status, wherein a readystatus indicates a readiness to receive data slices for storage; senddata slices to selected storage units in the ready status; and theprocessor further to direct delivery of data slices intended forselected storage units in a non-ready status to a temporary memory. 9.The distributed storage processing unit of claim 8, the processorfurther to: send data slices to the selected storage units only if anumber of selected storage units in the ready status is at least asgreat as a threshold number of data slices.
 10. The distributed storageprocessing unit of claim 9, the processor further to: discard dataslices associated with a particular selected storage unit from thetemporary memory in accordance with a discard policy.
 11. Thedistributed storage processing unit of claim 8, the processor furtherto: determine that a particular selected storage unit previously in anon-ready status has entered a ready status; and direct the movement ofdata slices intended for the particular selected storage unit from thetemporary memory into the particular selected storage unit.
 12. Thedistributed storage processing unit of claim 8, the processor furtherto: determine that a particular selected storage unit has remained in anon-ready status longer than a timeout threshold; select a replacementstorage unit; and direct the movement of data slices intended for theparticular selected storage unit from the temporary memory into thereplacement storage unit.
 13. The distributed storage processing unit ofclaim 8, the processor further to: direct data slices intended forselected storage units in a non-ready status to be stored in a temporarymemory of the distributed storage processing unit.
 14. The distributedstorage processing unit of claim 8, the processor further to: directdata slices intended for a particular selected storage unit in anon-ready status to be stored in a temporary memory of the particularselected storage unit.
 15. A distributed storage network comprising: aplurality of distributed storage units to receive data slices forstorage; a distributed storage processing unit to: create a plurality ofdata slices from a data segment, the plurality of data slices to bestored in selected distributed storage units; identify the selecteddistributed storage units, each of the selected storage units to storeparticular data slices of the plurality of data slices; determine whichof the selected distributed storage units is in a ready status, whereina ready status indicates a readiness to receive data slices for storage;send data slices to selected distributed storage units having a statusof ready; and direct delivery of data slices intended for selecteddistributed storage units having a status of non-ready to a temporarymemory.
 16. The distributed storage network of claim 15, furthercomprising: the distributed storage processing unit to send data slicesto the selected distributed storage units only if a number of selecteddistributed storage units having a status of ready is at least as greatas a threshold number of data slices.
 17. The distributed storagenetwork of claim 16, further comprising: the distributed storageprocessing unit to discard data slices associated with a particularselected storage unit from the temporary memory in accordance with adiscard policy.
 18. The distributed storage network of claim 15, furthercomprising: the distributed storage processing unit to determine that aparticular selected distributed storage unit previously in a non-readystatus has entered a ready status; and the distributed storageprocessing unit further to direct the movement of data slices intendedfor the particular selected distributed storage unit from the temporarymemory into the particular selected distributed storage unit.
 19. Thedistributed storage network of claim 15, further comprising: thedistributed storage processing unit to: determine that a particularselected distributed storage unit has remained in a non-ready statuslonger than a timeout threshold; select a replacement distributedstorage unit; and direct the movement of data slices intended for theparticular selected distributed storage unit from the temporary memoryinto the replacement distributed storage unit.
 20. The distributedstorage network of claim 15, further comprising: a local temporarymemory local to and associated with the distributed storage processingunit; and the distributed storage processing unit to direct data slicesintended for selected distributed storage units in a non-ready status tobe stored in the local temporary memory associated with the distributedstorage processing unit.
 21. The distributed storage network of claim15, further comprising: a local temporary memory local to and associatedwith a particular distributed storage unit in a non-ready status; andthe distributed storage processing unit to direct data slices intendedto be stored in the particular selected distributed storage unit, bestored temporarily in the local temporary memory of the particularselected distributed storage unit.